A simulation study comparing aberration detection algorithms for syndromic surveillance

BACKGROUND: The usefulness of syndromic surveillance for early outbreak detection depends in part on effective statistical aberration detection. However, few published studies have compared different detection algorithms on identical data. In the largest simulation study conducted to date, we compared the performance of six aberration detection algorithms on simulated outbreaks superimposed on authentic syndromic surveillance data. METHODS: We compared three control-chart-based statistics, two exponential weighted moving averages, and a generalized linear model. We simulated 310 unique outbreak signals, and added these to actual daily counts of four syndromes monitored by Public Health – Seattle and King County's syndromic surveillance system. We compared the sensitivity of the six algorithms at detecting these simulated outbreaks at a fixed alert rate of 0.01. RESULTS: Stratified by baseline or by outbreak distribution, duration, or size, the generalized linear model was more sensitive than the other algorithms and detected 54% (95% CI = 52%–56%) of the simulated epidemics when run at an alert rate of 0.01. However, all of the algorithms had poor sensitivity, particularly for outbreaks that did not begin with a surge of cases. CONCLUSION: When tested on county-level data aggregated across age groups, these algorithms often did not perform well in detecting signals other than large, rapid increases in case counts relative to baseline levels

An automated, broad-based, near real-time public health surveillance system using presentations to hospital Emergency Departments in New South Wales, Australia

BACKGROUND: In a climate of concern over bioterrorism threats and emergent diseases, public health authorities are trialling more timely surveillance systems. The 2003 Rugby World Cup (RWC) provided an opportunity to test the viability of a near real-time syndromic surveillance system in metropolitan Sydney, Australia. We describe the development and early results of this largely automated system that used data routinely collected in Emergency Departments (EDs). METHODS: Twelve of 49 EDs in the Sydney metropolitan area automatically transmitted surveillance data from their existing information systems to a central database in near real-time. Information captured for each ED visit included patient demographic details, presenting problem and nursing assessment entered as free-text at triage time, physician-assigned provisional diagnosis codes, and status at departure from the ED. Both diagnoses from the EDs and triage text were used to assign syndrome categories. The text information was automatically classified into one or more of 26 syndrome categories using automated "naïve Bayes" text categorisation techniques. Automated processes were used to analyse both diagnosis and free text-based syndrome data and to produce web-based statistical summaries for daily review. An adjusted cumulative sum (cusum) was used to assess the statistical significance of trends. RESULTS: During the RWC the system did not identify any major public health threats associated with the tournament, mass gatherings or the influx of visitors. This was consistent with evidence from other sources, although two known outbreaks were already in progress before the tournament. Limited baseline in early monitoring prevented the system from automatically identifying these ongoing outbreaks. Data capture was invisible to clinical staff in EDs and did not add to their workload. CONCLUSION: We have demonstrated the feasibility and potential utility of syndromic surveillance using routinely collected data from ED information systems. Key features of our system are its nil impact on clinical staff, and its use of statistical methods to assign syndrome categories based on clinical free text information. The system is ongoing, and has expanded to cover 30 EDs. Results of formal evaluations of both the technical efficiency and the public health impacts of the system will be described subsequently

Rapid detection of pandemic influenza in the presence of seasonal influenza

Background: Key to the control of pandemic influenza are surveillance systems that raise alarms rapidly and sensitively. In addition, they must minimise false alarms during a normal influenza season. We develop a method that uses historical syndromic influenza data from the existing surveillance system 'SERVIS' (Scottish Enhanced Respiratory Virus Infection Surveillance) for influenza-like illness (ILI) in Scotland. Methods: We develop an algorithm based on the weekly case ratio (WCR) of reported ILI cases to generate an alarm for pandemic influenza. From the seasonal influenza data from 13 Scottish health boards, we estimate the joint probability distribution of the country-level WCR and the number of health boards showing synchronous increases in reported influenza cases over the previous week. Pandemic cases are sampled with various case reporting rates from simulated pandemic influenza infections and overlaid with seasonal SERVIS data from 2001 to 2007. Using this combined time series we test our method for speed of detection, sensitivity and specificity. Also, the 2008-09 SERVIS ILI cases are used for testing detection performances of the three methods with a real pandemic data. Results: We compare our method, based on our simulation study, to the moving-average Cumulative Sums (Mov-Avg Cusum) and ILI rate threshold methods and find it to be more sensitive and rapid. For 1% case reporting and detection specificity of 95%, our method is 100% sensitive and has median detection time (MDT) of 4 weeks while the Mov-Avg Cusum and ILI rate threshold methods are, respectively, 97% and 100% sensitive with MDT of 5 weeks. At 99% specificity, our method remains 100% sensitive with MDT of 5 weeks. Although the threshold method maintains its sensitivity of 100% with MDT of 5 weeks, sensitivity of Mov-Avg Cusum declines to 92% with increased MDT of 6 weeks. For a two-fold decrease in the case reporting rate (0.5%) and 99% specificity, the WCR and threshold methods, respectively, have MDT of 5 and 6 weeks with both having sensitivity close to 100% while the Mov-Avg Cusum method can only manage sensitivity of 77% with MDT of 6 weeks. However, the WCR and Mov-Avg Cusum methods outperform the ILI threshold method by 1 week in retrospective detection of the 2009 pandemic in Scotland. Conclusions: While computationally and statistically simple to implement, the WCR algorithm is capable of raising alarms, rapidly and sensitively, for influenza pandemics against a background of seasonal influenza. Although the algorithm was developed using the SERVIS data, it has the capacity to be used at other geographic scales and for different disease systems where buying some early extra time is critical

Value of syndromic surveillance within the Armed Forces for early warning during a dengue fever outbreak in French Guiana in 2006

<p>Abstract</p> <p>Background</p> <p>A dengue fever outbreak occured in French Guiana in 2006. The objectives were to study the value of a syndromic surveillance system set up within the armed forces, compared to the traditional clinical surveillance system during this outbreak, to highlight issues involved in comparing military and civilian surveillance systems and to discuss the interest of syndromic surveillance for public health response.</p> <p>Methods</p> <p>Military syndromic surveillance allows the surveillance of suspected dengue fever cases among the 3,000 armed forces personnel. Within the same population, clinical surveillance uses several definition criteria for dengue fever cases, depending on the epidemiological situation. Civilian laboratory surveillance allows the surveillance of biologically confirmed cases, within the 200,000 inhabitants.</p> <p>Results</p> <p>It was shown that syndromic surveillance detected the dengue fever outbreak several weeks before clinical surveillance, allowing quick and effective enhancement of vector control within the armed forces. Syndromic surveillance was also found to have detected the outbreak before civilian laboratory surveillance.</p> <p>Conclusion</p> <p>Military syndromic surveillance allowed an early warning for this outbreak to be issued, enabling a quicker public health response by the armed forces. Civilian surveillance system has since introduced syndromic surveillance as part of its surveillance strategy. This should enable quicker public health responses in the future.</p

Disease surveillance using a hidden Markov model

<p>Abstract</p> <p>Background</p> <p>Routine surveillance of disease notification data can enable the early detection of localised disease outbreaks. Although hidden Markov models (HMMs) have been recognised as an appropriate method to model disease surveillance data, they have been rarely applied in public health practice. We aimed to develop and evaluate a simple flexible HMM for disease surveillance which is suitable for use with sparse small area count data and requires little baseline data.</p> <p>Methods</p> <p>A Bayesian HMM was designed to monitor routinely collected notifiable disease data that are aggregated by residential postcode. Semi-synthetic data were used to evaluate the algorithm and compare outbreak detection performance with the established Early Aberration Reporting System (EARS) algorithms and a negative binomial cusum.</p> <p>Results</p> <p>Algorithm performance varied according to the desired false alarm rate for surveillance. At false alarm rates around 0.05, the cusum-based algorithms provided the best overall outbreak detection performance, having similar sensitivity to the HMMs and a shorter average time to detection. At false alarm rates around 0.01, the HMM algorithms provided the best overall outbreak detection performance, having higher sensitivity than the cusum-based Methods and a generally shorter time to detection for larger outbreaks. Overall, the 14-day HMM had a significantly greater area under the receiver operator characteristic curve than the EARS C3 and 7-day negative binomial cusum algorithms.</p> <p>Conclusion</p> <p>Our findings suggest that the HMM provides an effective method for the surveillance of sparse small area notifiable disease data at low false alarm rates. Further investigations are required to evaluation algorithm performance across other diseases and surveillance contexts.</p

Public Health Emergency Preparedness and Response Communications with Health Care Providers: A Literature Review

<p>Abstract</p> <p>Background</p> <p>Health care providers (HCPs) play an important role in public health emergency preparedness and response (PHEPR) so need to be aware of public health threats and emergencies. To inform HCPs, public health issues PHEPR messages that provide guidelines and updates, and facilitate surveillance so HCPs will recognize and control communicable diseases, prevent excess deaths and mitigate suffering. Public health agencies need to know that the PHEPR messages sent to HCPs reach their target audience and are effective and informative. Public health agencies need to know that the PHEPR messages sent to HCPs reach their target audience and are effective and informative. We conducted a literature review to investigate the systems and tools used by public health to generate PHEPR communications to HCPs, and to identify specific characteristics of message delivery mechanisms and formats that may be associated with effective PHEPR communications.</p> <p>Methods</p> <p>A systematic review of peer- and non-peer-reviewed literature focused on the following questions: 1) What public health systems exist for communicating PHEPR messages from public health agencies to HCPs? 2) Have these systems been evaluated and, if yes, what criteria were used to evaluate these systems? 3) What have these evaluations discovered about characterizations of the most effective ways for public health agencies to communicate PHEPR messages to HCPs?</p> <p>Results</p> <p>We identified 25 systems or tools for communicating PHEPR messages from public health agencies to HCPs. Few articles assessed PHEPR communication systems or messaging methods or outcomes. Only one study compared the effectiveness of the delivery format, device or message itself. We also discovered that the potential is high for HCPs to experience "message overload" given redundancy of PHEPR messaging in multiple formats and/or through different delivery systems.</p> <p>Conclusions</p> <p>We found that detailed descriptions of PHEPR messaging from public health to HCPs are scarce in the literature and, even when available are rarely evaluated in any systematic fashion. To meet present-day and future information needs for emergency preparedness, more attention needs to be given to evaluating the effectiveness of these systems in a scientifically rigorous manner.</p

Online detection and quantification of epidemics

<p>Abstract</p> <p>Background</p> <p>Time series data are increasingly available in health care, especially for the purpose of disease surveillance. The analysis of such data has long used periodic regression models to detect outbreaks and estimate epidemic burdens. However, implementation of the method may be difficult due to lack of statistical expertise. No dedicated tool is available to perform and guide analyses.</p> <p>Results</p> <p>We developed an online computer application allowing analysis of epidemiologic time series. The system is available online at <url>http://www.u707.jussieu.fr/periodic_regression/</url>. The data is assumed to consist of a periodic baseline level and irregularly occurring epidemics. The program allows estimating the periodic baseline level and associated upper forecast limit. The latter defines a threshold for epidemic detection. The burden of an epidemic is defined as the cumulated signal in excess of the baseline estimate. The user is guided through the necessary choices for analysis. We illustrate the usage of the online epidemic analysis tool with two examples: the retrospective detection and quantification of excess pneumonia and influenza (P&I) mortality, and the prospective surveillance of gastrointestinal disease (diarrhoea).</p> <p>Conclusion</p> <p>The online application allows easy detection of special events in an epidemiologic time series and quantification of excess mortality/morbidity as a change from baseline. It should be a valuable tool for field and public health practitioners.</p

Probabilistic Daily ILI Syndromic Surveillance with a Spatio-Temporal Bayesian Hierarchical Model

BACKGROUND: For daily syndromic surveillance to be effective, an efficient and sensible algorithm would be expected to detect aberrations in influenza illness, and alert public health workers prior to any impending epidemic. This detection or alert surely contains uncertainty, and thus should be evaluated with a proper probabilistic measure. However, traditional monitoring mechanisms simply provide a binary alert, failing to adequately address this uncertainty. METHODS AND FINDINGS: Based on the Bayesian posterior probability of influenza-like illness (ILI) visits, the intensity of outbreak can be directly assessed. The numbers of daily emergency room ILI visits at five community hospitals in Taipei City during 2006-2007 were collected and fitted with a Bayesian hierarchical model containing meteorological factors such as temperature and vapor pressure, spatial interaction with conditional autoregressive structure, weekend and holiday effects, seasonality factors, and previous ILI visits. The proposed algorithm recommends an alert for action if the posterior probability is larger than 70%. External data from January to February of 2008 were retained for validation. The decision rule detects successfully the peak in the validation period. When comparing the posterior probability evaluation with the modified Cusum method, results show that the proposed method is able to detect the signals 1-2 days prior to the rise of ILI visits. CONCLUSIONS: This Bayesian hierarchical model not only constitutes a dynamic surveillance system but also constructs a stochastic evaluation of the need to call for alert. The monitoring mechanism provides earlier detection as well as a complementary tool for current surveillance programs

The Tell-Tale Heart: Population-Based Surveillance Reveals an Association of Rofecoxib and Celecoxib with Myocardial Infarction

Background. COX-2 selective inhibitors are associated with myocardial infarction (MI). We sought to determine whether population health monitoring would have revealed the effect of COX-2 inhibitors on population-level patterns of MI. Methodology/Principal Findings. We conducted a retrospective study of inpatients at two Boston hospitals, from January 1997 to March 2006. There was a population-level rise in the rate of MI that reached 52.0 MI-related hospitalizations per 100,000 (a two standard deviation exceedence) in January of 2000, eight months after the introduction of rofecoxib and one year after celecoxib. The exceedence vanished within one month of the withdrawal of rofecoxib. Trends in inpatient stay due to MI were tightly coupled to the rise and fall of prescriptions of COX-2 inhibitors, with an 18.5 % increase in inpatient stays for MI when both rofecoxib and celecoxib were on the market (P,0.001). For every million prescriptions of rofecoxib and celecoxib, there was a 0.5 % increase in MI (95%CI 0.1 to 0.9) explaining 50.3 % of the deviance in yearly variation of MI-related hospitalizations. There was a negative association between mean age at MI and volume of prescriptions for celecoxib and rofecoxib (Spearman correlation, 20.67, P,0.05). Conclusions/Significance. The strong relationship between prescribing and outcome time series supports a population-level impact of COX-2 inhibitors on MI incidence. Further, mean age at MI appears to have been lowered by use of these medications. Use of a population monitoring approach as an adjunct t

A Methodological Framework for the Evaluation of Syndromic Surveillance Systems: A Case Study of England

Background: Syndromic surveillance complements traditional public health surveillance by collecting and analysing health indicators in near real time. The rationale of syndromic surveillance is that it may detect health threats faster than traditional surveillance systems permitting more timely, and hence potentially more effective public health action. The effectiveness of syndromic surveillance largely relies on the methods used to detect aberrations. Very few studies have evaluated the performance of syndromic surveillance systems and consequently little is known about the types of events that such systems can and cannot detect. Methods: We introduce a framework for the evaluation of syndromic surveillance systems that can be used in any setting based upon the use of simulated scenarios. For a range of scenarios this allows the time and probability of to be determined and uncertainty is fully incorporated. In addition, we demonstrate how such a framework can model the benefits of increases in the number of centres reporting syndromic data and also determine the minimum size of outbreaks that can or cannot be detected. Here, we demonstrate its utility using simulations of national influenza outbreaks and localised outbreaks of cryptosporidiosis. Results: Influenza outbreaks are consistently detected with larger outbreaks being detected in a more timely manner. Small cryptosporidiosis outbreaks (<1000 symptomatic individuals) are unlikely to be detected. We also demonstrate the advantages of having multiple syndromic data streams (e.g. emergency attendance data, telephone helpline data, general practice consultation data) as different streams are able to detect different types outbreaks with different efficacy (e.g. emergency attendance data are useful for the detection of pandemic influenza but not for outbreaks of cryptosporidiosis). We also highlight that for any one disease, the utility of data streams may vary geographically, and that the detection ability of syndromic surveillance varies seasonally (e.g. an influenza outbreak starting in July is detected sooner than one starting later in the year). We argue that our framework constitutes a useful tool for public health emergency preparedness in multiple settings. Conclusions: The proposed framework allows the exhaustive evaluation of any syndromic surveillance system and constitutes a useful tool for emergency preparedness and response